Method and device for rendering text with combined typographical attributes for emphases to a computer display

ABSTRACT

The present disclosure relates to UI systems and processes including rendering of text in a combination of styles and sizes to a computer graphics display. Text may be rendered with one or more emphases to modify style, size, and weight according to emphases weightings and or relative positions of text to be emphasized.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR § 1.57.

SUMMARY

In some aspects, the techniques described herein relate to a method fordisplaying on a computer display information about electronic documentsmatching selection criteria and supporting user interaction therewith,the method including steps executed on one or more computer processorsof: running each electronic document through a Natural LanguageProcessing module to tokenize text of the electronic document and assignpart-of-speech tags and named entity tags to the tokenized text; runningeach electronic document's tokenized text, part-of-speech tags, andnamed entity tags through a summarization algorithm to create asummarization of the electronic document; computing a current bubblesize for each electronic document based on, at least in part, arelevancy score for the electronic document; and rendering on thecomputer display, for each electronic document, a summary of theelectronic document in a contiguous bubble region sized proportionatelyto the electronic document's current bubble size, the summary includingan image extracted from the electronic document, if any, and, accordingto threshold criteria: keywords selected from the tokenized text of theelectronic document if the electronic document's current bubble size isin a first range of thresholds of the threshold criteria, thesummarization of the electronic document if the electronic document'scurrent bubble size is in a second range of thresholds of the thresholdcriteria, or a full-text view of the electronic document if theelectronic document's current bubble size is greater than the secondrange of thresholds of the threshold criteria.

In some aspects, the techniques described herein relate to a methodwherein the keywords selected from the tokenized text of the electronicdocument are named entities. In some aspects, the techniques describedherein relate to a method wherein the electronic documents include oneor more of text documents, videos, images, web pages, social media, andcombinations thereof.

In some aspects, the techniques described herein relate to a methodwherein the full-text view of an electronic document that is a livenetworked electronic document includes an embedded window for the livenetworked electronic document. In some aspects, the techniques describedherein relate to a method, further including: establishing a proxyserver; and fetching the live networked electronic document via theproxy server. In some aspects, the techniques described herein relate toa method wherein the embedded window is established by means of aniframe.

In some aspects, the techniques described herein relate to a method,further including: optimizing arrangement of the contiguous bubbleregions on the computer display to be substantially non-overlapping andto display a substantially maximized quantity of the contiguous bubbleregions for a current zoom level, excepting that a focused contiguousbubble region may be permitted to overlap other contiguous bubbleregions.

In some aspects, the techniques described herein relate to a method,further including: accepting a user zoom gesture; determining a zoomlevel change in response to the user zoom gesture; scaling the currentbubble size for each electronic document in proportion to the zoom levelchange; and repeating the optimizing and rendering steps.

In some aspects, the techniques described herein relate to a method,further including: accepting a user zoom gesture; determining a zoomlevel change in response to the user zoom gesture; determining S, anumber of animation steps to smoothly animate the zoom level change;determining an ordered monotonic series of S bubble sizes for eachelectronic document between the electronic document's current bubblesize and a final bubble size that is in proportion to the current bubblesize and the zoom level change; and iterating S animations by settingeach electronic document's current bubble size to a next in theelectronic document's series of S bubble sizes and repeating theoptimizing and rendering steps.

In some aspects, the techniques described herein relate to a methodwherein the user zoom gesture is a focus gesture on a selected one ofthe contiguous bubble regions, wherein: designating the selected one ofthe contiguous bubble regions a focused contiguous bubble region;determining a zoom level change includes determining a zoom level thatwould result in a full-text view of the focused contiguous bubble regionwhen the animations are complete; iterating S animations includessmoothly panning the contiguous bubble regions such that the focusedcontiguous bubble region is fully and centrally viewable; and the useris permitted interaction with the full-text view of the focusedcontiguous bubble region.

In some aspects, the techniques described herein relate to a methodwherein the permitted interaction is one of scrolling or selecting ahyperlink.

In some aspects, the techniques described herein relate to a methodwherein if an iteration step would cause an electronic document's bubblesize to satisfy different threshold criteria than the electronicdocument's current bubble size, the rendering associated with thatiteration step includes a smooth transition between keywords andsummarization or summarization and full-text view.

In some aspects, the techniques described herein relate to a methodwherein a quantity and sizes of keywords displayed in an electronicdocument's contiguous bubble region are optimized to fill availablespace in the contiguous bubble region.

In aspects of certain embodiments, the techniques described hereinrelate to a method for maintaining a position on a computer display of atarget word in a body of text subject to modifications, the displaydriven by a rendering engine, the method including: determining aninitial coordinate position of the target word; dividing the body oftext into an ordered set of words; calculating a width and a height ofeach word as the word would be rendered by the rendering engine applyingstyle information applicable to the word; forward rendering, includingstarting at the target word and the target word's coordinate position,rendering in sequence each word, adding a line break preceding the wordif the word would not fit on a current line based on the word'scalculated width and height; backward rendering, including starting atthe target word and the target word's position, rendering backward eachprior word before the target word in reverse sequence, adding a linebreak following the word if the word would not fit on a current linebased on the word's calculated width and height; if the body of text ismodified by an addition or deletion of words, repeating the forwardrendering and backward rendering steps; and if the body of text ismodified in a manner affecting the width or height of one or more words,repeating the dividing, calculating, forward rendering, and backwardrendering steps.

In aspects of certain embodiments, the techniques described hereinrelate to a method for maintaining a position on a computer display of atarget word in a body of text subject to modifications, the displaydriven by a rendering engine, the method including: determining aninitial coordinate position of the target word; dividing the body oftext into an ordered set of words; calculating a width and a height ofeach word as the word would be rendered by the rendering engine applyingstyle information applicable to the word; forward rendering, includingstarting at the target word and the target word's coordinate position,rendering in sequence each word unless the word would not fit on acurrent line based on the word's calculated width and height, whereuponbreaking the word into a leading portion and a trailing portion suchthat the leading portion fits on the current line, rendering the leadingportion, rendering a line break, and rendering the trailing portion if;backward rendering, including starting at the target word and the targetword's position, rendering backward in reverse sequence each prior wordbefore the target word unless the word would not fit on a current linebased on the word's calculated width and height, whereupon breaking theword into a leading portion and a trailing portion such that thetrailing portion fits on the line, rendering the trailing portion,rendering a line break, and rendering the leading portion; if the bodyof text is modified by an addition or deletion of words, repeating theforward rendering and backward rendering steps; and if the body of textis modified in a manner affecting the width or height of one or morewords, repeating the dividing, calculating, forward rendering, andbackward rendering steps.

In some aspects, the techniques described herein relate to a methodwherein rendering of a word that will be broken is effected by renderingthe word to be split on both the current line and a next line, in therendering direction, subject to cropping boxes which cut off thetrailing or leading portions or the word.

In aspects of certain embodiments, the techniques described hereinrelate to a method for rendering of text in a combination of styles andsizes to a computer graphics display with a rendering engine, the methodincluding steps of: running the text through a Natural LanguageProcessing module to break the text into word or character tokens, eachtoken having an associated sequential index according to its order inthe text; instantiating a set of Emphases, where each Emphasis specifiesa parameter for styling of text and a position index or range ofpositions indicating how the Emphasis affects the rendering of tokens inthe text, wherein multiple Emphases may affect any given token and theEmphases may overlap; combining topographical attributes of multipleEmphases by a weighted average, a linear combination of attributes, or amathematical transformation; and mapping the set of weighted Emphases tothe style, size, color and other typographical attributes of the textcorresponding to each token by combining relevant Emphases styles.

In some aspects, the techniques described herein relate to a method,wherein mapping the set of weighted Emphases is performed by specifyinga range including a start position and an end position, and a fixedweight for each Emphasis to the text corresponding to the tokens withinthe specified range.

In some aspects, the techniques described herein relate to a method,wherein mapping the set of Emphases includes specifying for eachEmphasis a weight, a token index for a center, and a standard deviation,and then calculating the weight for each Emphasis for a token using adistance from the token index to the Emphasis center and computing aweight using a gaussian decay function with the standard deviation.

In some aspects, the techniques described herein relate to a methodwherein the weight, position, or other attribute of one or more Emphasesis incrementally modified to produce a series of renderings of the textwhich may be displayed as a smooth animation or manipulation of thetext.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1D illustrate portions of display captures of exemplarydocument renderings of varying levels of detail in accordance withembodiments of the invention with FIG. 1A illustrating image only, FIG.1B illustrating image with keywords, FIG. 1C illustrating image withsummary, and FIG. 1D illustrating image with full text.

FIG. 2 illustrates an exemplary method for rendering text and images atmultiple levels of detail in accordance with an embodiment of theinvention.

FIGS. 3A-3F illustrate an exemplary user interface displayinginformation about Documents at increasing levels of zoom and detail.

FIG. 4 illustrates an exemplary method for document topic assignment inaccordance with an embodiment of the invention.

FIG. 5 illustrates a exemplary method of graph layout and display inaccordance with an embodiment of the invention.

FIG. 6 illustrates an exemplary method for culling and deferredrendering optimization in accordance with an embodiment of theinvention.

FIG. 7 illustrates exemplary text which has been rendered usingNLP-based Emphasis in accordance with an embodiment of the invention.

FIG. 8 illustrates a sample of text on which Emphasis has been appliedat the word level in accordance with an embodiment of the invention.

FIG. 9 illustrates an exemplary computing and network environment foraspects of the invention.

DETAILED DESCRIPTION

As used in this description, a Document is defined as an artifact ofinformation, such as a text document, a video, an image, a web page,social media comment, or some combination of thereof. A Document mayhave multiple Variants, such as translations from the original languageinto other languages.

As used in this description, a Variant is an alternative form of aDocument. For example, a document of written text may have one Variantwith the text in the original English, and another Variant with atranslation of the text to Arabic.

In some scenarios, a Document is displayed in a contiguous region thatmay be circular, semi-circular, or generally rectangular. The shape inwhich a Document is rendered in some form may be referred to as aBubble.

I. Rendering document with multiple levels of detail

I.1 Overview

In accordance with an embodiment, a Document may be a body of text, avideo, an image, a social media comment, etc. A Document may be renderedinto a Bubble—a contiguous region that may be circular orrectangular—such as Bubbles 11 in FIGS. 1A-1D. As a Bubble is zoomed itbecomes larger to fill more and more of the available screen area. Inaccordance with an embodiment, increasing screen area of a Bubble isused to display more of a Document in a variety of ways outlined below.

I.2 Image Only View

A Document often has an associated image. This image may be a photographassociated with a news article, or a still frame from a video, or anactive video. Referring to FIG. 1A, when a Bubble 11 is zoomed out farenough that there is not enough space to render text, the image or video12 can be rendered as large as possible within the Bubble. This allows auser to visually obtain clues about the content of the Document.

I.3 Keywords View

Referring to FIG. 1B, when enough screen area is available in Bubble 11to render some text, the relative size of image 12 may be reduced into asubregion of the Bubble 11—the top of the Bubble for instance—withDocument keywords 13 displayed in a separate sub region—below forinstance. The keywords may be obtained by any number of techniques. Inone embodiment, a statistical or neural Named Entity Recognition (NER)model is applied to obtain and identify keywords in the text such asperson names, placenames, geographic points of interest, currency, andother classes. In certain embodiments top keywords 13 obtained by NER orother methods are displayed as illustrated in FIG. 1B. Additionally,Document information such as Document time 14 and Document sourceidentifier 15 may be displayed in Bubble 11.

I.4 Document Summary View

Referring to FIG. 1C, when Bubble 11 is enlarged enough to legiblydisplay longer segments of text, keywords 13 as illustrated in FIG. 1Bare replaced with artificial sentences 16 that summarize the meaning ofthe entire text. This encompasses the field of Document Summarization inmachine learning. A statistical or neural network model is used tocreate the summarization.

Summarization techniques fall in two categories: abstractive andextractive. Use of extractive summarization permits gradually expandingof the summary until the full text is visible in a gradual transition.

I.5 Full-Text View

Referring to FIG. 1D, when Bubble 11 is large enough to legibly renderfull text of a Document, then the Document text is rendered in ascrollable sub-region 17 of Bubble 11—below the image or video, forexample. In this mode, pan and zoom controls may be suspended to allowthe user to concentrate on the given Document, and the user can scrollthrough the full text using, for example, scrollbar 18 or mouse ortouchscreen gestures. The text may be rendered with emphasis inaccordance with NLP tags, as described in a later section.

I.6 HTML Web Page View

Where a Document originates from or is associated with a World Wide Web(WWW) URL, an alternate to Full-Text View can be provided where insteadof simply providing the text of a Document, the associated web page canbe rendered seamlessly inside a Bubble.

This is made possible by the “iframe” HTML tag which allows a webbrowser to render a web page within another web page, with somerestrictions. In certain embodiments, a web “proxy”, which is a webserver URL, fetches web pages on behalf of the client to bypass somerestrictions.

The iframe has no knowledge of the location or size of the enclosingBubble. It may be necessary to scale and position the iframe within theBubble on a frame-by-frame basis to maintain the illusion that the webpage is embedded firmly in the Bubble. This can be accomplished by usingCSS “transform” directives.

I.7 Smooth Transitions

While the user is “zooming” onto a Document Bubble or set of DocumentBubbles, smooth transitions are made between the levels of detail.Images at the image-only phase are smoothly morphed into their newposition and relative size. NER Keywords meld into summary text and theninto full text smoothly using techniques described later. The shape ofthe Bubble can also morph smoothly from a circle when small, to arectangle with rounded corners when large and focused.

I.8 Method for Rendering Text and Images at Multiple Levels of Detail

With reference to FIG. 2 , the method starts with Document 201 whichconsists of text and optionally an image. First the Document text is runthrough Natural Language Processing module 202 to divide it into tokens(in most cases a token is a word), analyze the text to assignpart-of-speech tags and named entity tags.

The output of the NLP process is a list of tagged tokens 203 which arethen fed through document summarization algorithm 204. The summarizationalgorithm, one of many well known in the field, determines which partsof the text best summarize the meaning and identifies the tokens for thesummarization 205.

The method then makes a series of decisions of what tokens to include inthe display Bubble based on the display size of the representation onscreen of the Document (the Bubble as defined previously). In steps206-209, these decisions decide the content of the Bubble according todisplay size thresholds. The thresholds may be variable or constant. Aratio of the display width to the font size of the display can be usedto determine such thresholds. In certain embodiments, the thresholds maybe identified as:

NER_THRES—minimum threshold size needed to display tokens with selectNER tags;

POS_THRES—minimum threshold size needed to display tokens with selectPOS tags;

SUM_THRES—minimum threshold size needed to display top documentsummarization extracts; and

FULL_THRES—minimum threshold to display all tokens.

At the end of this process there is a list of tokens and an image todisplay in a Bubble.

II. Rendering multiple documents in view

II.1 Overview

With reference to FIGS. 3A-3F, the user is presented with a userinterface 110 including a search bar 115, filtering controls 116, and aset of rendered Bubbles, Bubbles 112 and 113, for example, representingDocuments. This provides the user with an environment in which she canexplore the content. The Documents presented may represent, for example,the top 50 documents matching a search query against a database ofmillions of documents. Or the documents may represent a set of livevideos of a sporting event that the user may arrange to best view theevent. A user may use keyboard, mouse or touch gestures to zoom and panthe content. FIGS. 3A-3F are ordered in increasing zoom levels and, ifrendered from a single zoom or focus gesture, would represent selectframes of a smooth transition animation of the zoom.

II.2 Topic Classification

In some embodiments, it is useful to place Documents that are similarnear each other in the multiple document layout. This allows the user tofind and browse an area of related documents that are of interest toher.

There are many dimensionality reducing techniques in the literature thatcould be used. In some embodiments, the option used is dividing theDocuments into clusters, where each cluster represents a “topic”.Documents in the same topic can be about similar subject matter.

Most applicable reducing techniques involve processing the Document intoa single vector of real numbers, called an “embedding”. There are manytechniques in the literature for generating document embeddings fromtext. In some embodiments a model known as the Universal SentenceEncoder (USE) is used. In other embodiments a Bidirectional EncoderRepresentations from Transforms (BERT) model is used on Document text,transcript, keywords, or headlines depending on the source.

BERT or USE or other techniques represents each Document as a vector ofreal numbers, known in the field as an “embedding”.

To map embedding vectors for each Document into discrete clusters, manyclustering techniques can be used. In some embodiments a GaussianMixture Model (GMM) is used. In others, K-means is used; both are welldocumented in the field. These techniques assign each Document to atopic cluster.

II.2.1 Method of Document Topic Assignment

FIG. 4 illustrates a method for the clustering of documents into anumber of “topics,” starting with a set of Documents 401 to ingest.

For each Document, text is extracted 402 on which the embedding is to bebased. Extraction of text is relatively straightforward but varies bythe Document type and origin. Valid text extraction techniques includethe use of the Document title or headline (if it has one), the completebody of the Document, or a summarization of the Document.

Once the text is selected, it is run through an Encoder 403 described inthe previous section. It generates a semantic vector, also known in thefield as an “embedding” 404.

The embeddings are then stored in a database 405 associated with theoriginal Documents. If desired, it is possible to omit the storage inthe database to process a set of Documents after the search isperformed, some of which may not have seen before. It is desirable togenerate the embeddings before a search is performed because it iscomputationally expensive. However, it is not always possible to haveprocessed the search results beforehand.

Assuming storage of the embeddings in the database, when the usersearches 406 the database 405, both the matching documents and theirembeddings 407 are retrieved.

Then the high dimensional embeddings are run through a clusteringalgorithm 408 to group “similar” Documents into a topic (each topic isassigned a number, or id). This is discussed in more technical detail inthe prior section. The result is a topic id associated with eachDocument.

II.3 Optimized Layout

The layout algorithm takes many Bubbles, each tagged with a topiccategory, and produces a layout in which the Bubbles for similarDocuments are, in general, placed near each other. This is achieved withan iteratively optimized layout. In addition to the initial iterativelayout, the layout will smoothly adapt to changes in the scene, such asthe insertion of a new Bubble or a removal of a Bubble.

II.3.1 Force Directed Graph Layout

To layout Bubbles, each with an associated Topic, a Force Directed Graphis used. A Force Directed Graph consists of “nodes” and “links” betweenthem. In certain embodiments, “nodes” are Bubbles. Bubbles with the sameTopic number are connected together with a “link”. The Force DirectedGraph can utilize a few physics modeling rules whereby nodes areattracted to each other, but collisions between nodes are repulsed.Nodes with links may be more attracted to each other.

The Force Directed Graph may be an iterative process. The layout startsin a mostly random state, and then the graph rules for attraction andrepulsion are calculated and the Bubbles are moved into place, such thatBubbles with the same topic are typically neighbors to each other.

If the size of the display window changes, or Bubbles are added orremoved due to filtering, then the Force Directed Graph algorithmrecomputes the layout and the iterative change to position is displayedas an animation to the user.

II.3.2 Method of Graph Layout and Display.

As illustrated in FIG. 5 , a method graph layout and display isprovided, starting with a set of Documents with associated topics 501.See section I.2 “Topic Classification” for more details. A “size” isassociated with each Document. This is the display size radius of theDocument on the screen. It can be based on search relevancy scores,sentiment analysis, or any other metric.

Model Parameters are initialized in 502. All documents of the same topicare fully linked together with an attractive force. All documents of thesame topic are also assigned the same random position for that topic.This initializes the documents of the same topic into the same initialpositions, which helps hold them together during the graph iterations.Finally, the size parameters are initialized for each document. Thisdetermines the on-screen size of each Bubble and informs the iterativelayout process to avoid collisions between bubbles.

At step 503, the force directed graph model is called for a singleiteration, during which it evaluates the forces imposed on the nodes(documents) by mutual attraction, link attraction, and collisionrepulsion. From this iteration an updated data structure of nodepositions is obtained, with each node associated with one document.

While still iterating, the display of the Bubbles is updated on screenaccording to the new node positions at 504. This allows the user to seethe bubbles gradually move into their final positions, or adapt to newfiltering.

At the end of each iteration, the layout error is checked if it is belowa threshold. If the error meets the threshold, the iterations are ended;otherwise additional iterations are run.

II.4 Color and Size

The background color of each Bubble can be set according to the topic ofthat Bubble's Document. Each topic may be assigned a color, and eachBubble of that topic uses the assigned color in, for example, itsbackground color, outline or other manner.

In some embodiments, color can be used for other purposes such ashighlighting age or source or language of the Document.

Size of the Bubbles can be used in many ways. In one embodiment, size isused to indicate relevancy to the search just performed in which Bubblesrepresenting more relevant Documents are larger. In other instances,size may indicate age, length, or popularity. The Force Directed Graphtakes into account the size of the Bubbles during layout.

II.5 Culling and Deferred Rendering Optimizations

Rendering large numbers of feature-rich Bubbles is taxing even on amodern web browser. According to embodiments of the invention, twotechniques make it manageable:

First, Bubbles which are off-screen may be “culled”. Such Bubbles arenot rendered since they are not visible.

Second, an inventive “deferred rendering” technique is used. Duringintensive zoom in and out operations, all Bubbles are needed to updatetheir display on each frame. This is too intensive for many browsers. Soinstead, during the movement of a zoom, only the focused Bubble isupdated, and the remaining bubbles are updated when the motion ends.This technique combined with a transformation applied to the deferredBubbles to scale their content, the motion appears fluid until theactual zoom update.

II.5.1 Method for Culling and Deferred Rendering Optimization

FIG. 6 illustrates an exemplary method for culling and deferredrendering optimization. Each Bubble is represented in the view graph asa “Component” 601. For each Component, the logic depicted in FIG. 6 isperformed.

If the area extents of the Component are out of the bounds of theviewport as determined at step 602—i.e., the Component would be renderedentirely off-screen—then the elements of the Component are removed fromthe Document Object Model (DOM) at step 605.

If there is movement underway, a pan or zoom manipulation for example,then many Components will need to update their state and their DOMrepresentation. Often more work needs to be done than can beaccomplished in a single frame, causing uneven rendering or a delayedresponse. To avoid these performance issues, updates of Components thatare not currently “focused”—i.e., not selected by the user—are deferredat steps 603. Otherwise, step 606 updates the Component and step 607renders the update to the DOM.

A deferred Component is still displayed on the screen, and its DOMelements remain in the DOM, however it is not actively updated. The onlyupdate that occurs is an adjustment to the Component's Cascading StyleSheet (CSS) transformation which is adjusted for the pan and zoom changein steps 604. This preserves the illusion of a Component moving throughspace as a solid body, but the details of the rendering are not updateduntil the motion is complete.

In an alternative embodiment, the same steps as above are taken, exceptnon-focused Components are allowed to update in incremental groupssufficiently small to avoid the performance issues and maintain smoothframerates.

III. Focus and Navigation

III.1 Navigation

The Bubbles represent the Documents at multiple levels of detail, asdescribed previously. The user pans and zooms on the display of multipleBubbles, and zooms in on those of interest. As the screen area availableto a Bubble increases, it displays more detail.

Alternatively, the system can be configured to allow the user to zoom inon a single Bubble without zooming the entire multi-document display. Asingle Bubble can hover above neighboring Bubbles, or neighboringBubbles can move out of the way of an enlarged focused Bubble using alayout optimization algorithm.

The user pans the display by clicking and dragging the mouse. On a touchscreen the user pans simply by dragging the scene around with herfinger. With a mouse, the user can zoom using the scroll wheel. On atouch screen device such as a tablet, the user can zoom using a“pinch-to-zoom” gesture. With a mouse or a touch screen the user can doa fast zoom to a particular Bubble by double-clicking on the Bubble ofinterest.

III.2 Focus

At any given time, a single Bubble can have “Focus”. A user focuses aBubble by clicking on it. A Focused Bubble can be visually indicated,for example, by rendering its border white. A Focused Bubble hasrendering priority (see prior sections on deferred rendering). Focusinga Bubble will often load additional information about that Document intothat Bubble, including, for example, the actual video stream. In somemodes, only a focused Bubble can play audio and video.

III.3 Single Focus View

When the user wants to simply read the full text of the Document, theymay zoom into the Focused Bubble into a “Single Focus View”. This viewmay be triggered by double-clicking a Bubble. The Bubble zooms to occupya substantial portion of the viewing window, in some embodimentsperfectly matching the size of the viewing window. Other Bubbles stillvisible outside of the focused Bubble may be faded. Normal pan/zoomnavigation mouse and touch gestures may be disabled, allowing the userto scroll the text with a standard scroll wheel and to control play of avideo, if any.

Single Focus View may be exited by again double-clicking the focusedBubble, or clicking anywhere outside of the focused Bubble.

FIGS. 3A-3F represent exemplary animation frames from a rendering of azoom to Bubble 112, beginning with a relatively small rendering ofBubble 112 in FIG. 3A and ending in a full-text view in Bubble 112 inFIG. 3F. Such zoom may be in response to a fast zoom, a focus, or singlefocus indication by the user.

III.4 Focus Tracking and Exposure

The amount of time a user spends focused to a high level of zoom on aDocument may be tracked. In some embodiments, if the time exceeds athreshold, then the Bubble may be “dimmed”—i.e., an effect to gray outand reduce the contrast of the Bubble. In this way, as the user movesfrom looking in detail at Bubble to Bubble, she will leave a trail of“dimmed” bubbles and will easily be able to see what she has read.

The present inventor has disclosed similarly useful techniques in U.S.patent application Ser. No. 12/603,767, filed Oct. 22, 2009, which ishereby incorporated in its entirety for all purposes.

IV. Sophisticated Rendering of Text

IV.1 Emphasis based on Natural Language Processing

When text is rendered, the size can be modulated on a word-by-word basisto reflect its importance. More critical words can be emphasized moreheavily with a larger font or bolder font attributes or othercharacteristics. The benefit of this emphasis-based font rendering isthat the text is easier to read quickly or skim. It is easier to findthe parts of the Document that the reader is actually interested in.

To determine the importance of each word and how it should be rendered,Natural Language Processing (NLP) can be used. NLP is used to getParts-of-Speech (POS) tags such as “Proper Noun”, “Adjective”, “Verb”.NLP is also used for Named Entity Recognition (NER). For each word orphrase, NER will identify it as a “Geographic Placename”, “Person”,“Institution”, etc. NLP can be implemented in a variety of ways, fromolder rule-based systems to neural networks. In some embodiments,statistical NLP models are used, but the technique can be used with anyNLP process.

In addition to NER and POS tagging, the NLP process “tokenizes” thetext—that is, it splits the continuous string of text into a list ofwords or parts of words called tokens. The NER and POS tags areassociated with tokens.

When the text is rendered, the tokens are reassembled, assigningemphasis on each token based on the part of speech or presence of anamed entity. Proper Nouns like “Detroit”, for example, are emphasizedmore than the article “The” or conjunction “and”.

FIG. 7 illustrates text which has been rendered using NLP-based Emphasisto increase font size of proper nouns, e.g., “Harvard” 71 and “MIT” 72,and decrease the size of words such as “said” 73 and “and” 74.

IV.2 Combining sources of Emphasis and Text Fragments

When rendering NLP processed text it is often desirable to combine manyfactors of emphasis and styling into one layout and rendering. Forexample, it may be desirable to enlarge the words around a keyword toprovide context, while also enlarging and changing the color of apossibly overlapping sentence, while “pushing” the default text to besmall and not visible. It is further desirable to smoothly adjust thesecompeting styles over periods of time; for example, removing theemphasized sentence over several seconds.

In one embodiment, the text if first run through Natural LanguageProcessing which breaks the characters of the text into tokens (roughly,words), and sentences. Each token has a sequential index associated withit.

One unit of styling is called an “Emphasis”. An Emphasis specifies alocation in the list of NLP tokens that make up the body of the text. Alocation is specified as a token index, or a continuous range ofindices, or a central index for a decay function. A decay function candetermine how the effect of styling from a particular Emphasis isdiminished for tokens farther from the center index. In someembodiments, a gaussian function with a standard deviation in indexunits defining the rapidity of decay is used.

An Emphasis can include not just location but can also define thestyling for the rendered text. The style information can be font sizemultipliers, foreground or background color, aspect ratio, visibility,or any other attribute commonly associated with text rendering. EachEmphasis does not need to specify all possible attributes.

A special “visibility” attribute may be used in conjunction with aglobal “visibility threshold”. Visibility can be weighted and decayedlike any other style attribute, but if it falls below the visibilitythreshold for a particular token index then that token is not rendered.Adjustment of the visibility threshold allows an easy way to control howmuch information is displayed in a fixed area.

An Emphasis may also include a “weight”. When multiple Emphases overlapa particular token index location with the same style attribute (such assize), the style specified from each Emphasis is combined using aweighted mean of the Emphases. The weight is optionally decayed using adecay function as described above.

The “Layout Engine” generates an HTML Document Object Model (DOM) forthe Hypertext Markup Language (HTML) containing the exact position andstyle attributes of the text for rendering by a web browser or otherHTML rendering engine. The technique is readily applicable to other textrendering environments.

A list of multiple Emphases is provided to the Layout Engine along withthe tokenized text. For each token, the Layout Engine retrieves a listof Emphases affecting that location, computes the weighted mean of anyconflicting styles, and applies that style to the token. Styles such assize also affect the rendering and positioning of text.

One of the benefits of this Emphasis system is that changes to Emphasiscan be easily and smoothly applied. For example, to shrink a sentencefrom the rendering smoothly over the course of a few seconds a singleEmphasis just needs to adjust its weight and font size multiplier. Thereis no need to run through the entire text attempting to tweak fontattributes before running the Layout Engine.

An optimization may be necessary to lookup Emphases quickly that matchthe location index for each token. This can be expensive for the LayoutEngine to attempt to apply all Emphases for each index. To improve thisperformance dramatically, a lookup table can be created which maps indexlocations to lists of all relevant Emphases.

IV.3 Focus Word Bidirectional Layout

FIG. 8 illustrates sample text on which Emphasis has been applied at theword level. The user is perhaps interested in more detail around theword “Iraqi” 81. This can be accomplished by increasing the sizeemphasis for the text before and after the word Iraqi. However, in atraditional text layout engine—such as a web browser—once the textbefore the word “Iraqi” is modified, then the position of the word“Iraqi” will be lost; it will either be rendered before or after itsprevious position.

It can be desirable to allow the user to “zoom in” on a word and getmore contextual information. If the word position is lost, then the userwill be disoriented and the effect is lost. This is especially true whenmanipulating heavily emphasized text as described in the previoussections.

To solve this problem, an inventive bidirectional layout algorithm isapplied. Such an algorithm starts with a “focus word” at a fixedposition. Like a conventional layout engine, the algorithm placesfollowing words in succession until the text box is full or there is notext remaining. However, the algorithm then returns to the initial focusword and, starting from the focus position, it places previous words inbackwards order, including optional word breaks at the line edges.

This bidirectional layout mechanism allows user interface experiencessuch as: a user positioning their mouse over a word, using the scrollwheel to “zoom in” on the word, and have the text layout responsivelyincrease the detail around that word while leaving the word under themouse pointer.

IV.4 Sub-Character Line Breaks for Smooth Text Transitions

Smooth transitions between text layouts is also a concern. When a userresizes a text area or commands a gradual change to the formatting of alaid-out text, then the transition should be fluid and not disorientingto the user.

One source of disorientation is the use of word-breaks. Word breakscause a word to wrap to the next line if it does not fit on theremainder of the current line. If a user changes the size of a window oftext with word-breaks enabled, then the change in layout is abrupt anddistracting. This can be improved by turning off word-breaks. Mostcurrent layout engines, such as that found in a modern web browser, willbreak the words at the granularity of a single letter or character.Unfortunately, that is not enough to fix the jarring experience.

An inventive Layout Engine instead can be configured to break words atthe sub-character pixel level. This allows smooth transitions betweenlayouts that are not disorienting—especially when combined with theFocus Word Bidirectional Layout described earlier.

FIG. 8 illustrates exemplary sub-character line breaks such as the “D”82A and 82B in “Dubai” and the “y” 83A and 83B in “Syria.”

Words broken at the pixel level, however, can be difficult to read. Thisis addressed by using a timed transition when the user is idle. Afterthe user's command (window resize, refocus, etc.) is complete and thetransition is done, and the user is idle for a certain time, e.g., oneto three seconds, the words are shifted back to word-break in their newpositions, using a smooth transition over half a second or so.

V. Computing and Network Environment

With reference to FIG. 9 , aspects of an exemplary computing and networkenvironment of the invention are illustrated. For example, a computerplatform 800 for user interaction is provided. Computer platform 800could be, by way of example, a desk-top computer, a tablet-stylecomputer, a laptop, or a smart phone. Computer platform 800 includes adisplay 810, a processor 850, storage 860, a network interface 870, andinput devices such as keyboard 820, mouse 830, or touch screen 840.Processor 850, storage 860 and other computer platform elements may beinterconnected by bus 890. Storage 860 may optionally include a storageslot in which removable storage devices may be inserted. The computerplatform is configured with processor executable instructions accessedfrom storage 860 for carrying out interaction with the user andretrieval and storage of information. Information may be stored viastorage/storage slot 860 or may be stored on a network accessibleresource such as file storage server 950 and data store 930. Networkinterface 870 may be a wire-line or wireless network interface.

Computer platform 800 may be connected via network interface 870 tonetwork 920. Network 920 may be the internet. Network 920 allowscomputer platform 800 to communicate with, for example, web platform900, other servers 910, and social media server 940. Also, network 920permits communication with search provider 915. Provider facility 915may include a server 960 connected with network 920, code storage 970and document databases 980, 990 and 965.

Data and computer processor instructions for carrying the inventivemethods may be stored on computer platform 800 in, e.g., storage/storageslot 860, or at search provider facility server 960. Processor 850,servers 900, 940, or 960 may carry out one or more steps of theinventive methods.

What is claimed is:
 1. A method for rendering of text in a combinationof styles and sizes to a computer graphics display with a renderingengine, the method comprising steps of: running the text through aNatural Language Processing module to break the text into word orcharacter tokens, each token having an associated sequential indexaccording to its order in the text; instantiating a set of Emphases,where each Emphasis specifies a parameter for styling of text and aposition index or range of positions indicating how the Emphasis affectsthe rendering of tokens in the text, wherein multiple Emphases mayaffect any given token and the Emphases may overlap; combiningtypographical attributes of multiple Emphases by a weighted average, alinear combination of attributes, or a mathematical transformation;mapping the set of weighted Emphases to the typographical attributes ofthe text corresponding to each token by combining relevant Emphasesstyles; and rendering, using the rendering engine, each token accordingto its typographical attributes to the graphics display.
 2. The methodas claimed in claim 1, wherein mapping the set of weighted Emphases isperformed by specifying a range comprising a start position and an endposition, and a fixed weight for each Emphasis to the text correspondingto the tokens within the specified range.
 3. The method as claimed inclaim 1, wherein mapping the set of Emphases includes specifying foreach Emphasis a weight, a token index for a center, and a standarddeviation, and then calculating the weight for each Emphasis for a tokenusing a distance from the token index to the Emphasis center andcomputing a weight using a gaussian decay function with the standarddeviation.
 4. The method according to claim 1, wherein an attribute ofone or more Emphases is incrementally modified to produce a series ofrenderings of the text which may be displayed as a smooth animation ormanipulation of the text.
 5. The method according to claim 4, whereinthe incrementally modified attribute is one of weight, position, andcolor.
 6. The method according to claim 1, wherein typographicalattributes include at least style, size, and color.
 7. A computingdevice comprising: at least one processor; a graphics display; and acomputer-readable medium having encoded thereon computer-executableinstructions to cause the at least one processor to perform operationscomprising: running a text through a Natural Language Processing moduleto break the text into word or character tokens, each token having anassociated sequential index according to its order in the text;instantiating a set of Emphases, where each Emphasis specifies aparameter for styling of text and a position index or range of positionsindicating how the Emphasis affects the rendering of tokens in the text,wherein multiple Emphases may affect any given token and the Emphasesmay overlap; combining typographical attributes of multiple Emphases bya weighted average, a linear combination of attributes, or amathematical transformation; mapping the set of weighted Emphases to thetypographical attributes of the text corresponding to each token bycombining relevant Emphases styles; and rendering, using a renderingengine, each token according to its typographical attributes to thegraphics display.
 8. The computing device as claimed in claim 7, whereinmapping the set of weighted Emphases is performed by specifying a rangecomprising a start position and an end position, and a fixed weight foreach Emphasis to the text corresponding to the tokens within thespecified range.
 9. The computing device as claimed in claim 7, whereinmapping the set of Emphases includes specifying for each Emphasis aweight, a token index for a center, and a standard deviation, and thencalculating the weight for each Emphasis for a token using a distancefrom the token index to the Emphasis center and computing a weight usinga gaussian decay function with the standard deviation.
 10. The computingdevice according to claim 7, wherein the weight, position, or otherattribute of one or more Emphases is incrementally modified to produce aseries of renderings of the text which may be displayed as a smoothanimation or manipulation of the text.
 11. The computing deviceaccording to claim 10, wherein the incrementally modified attribute isone of weight, position, and color.
 12. The computing device accordingto claim 7, wherein typographical attributes include at least style,size, and color.